SherbyNet - /page/wikibook/entry.mu

Micron Document

<!DOCTYPE html>

<title>Data parallelism</title>

</head>

<span id="openzim-page-title" class="mw-page-title-main"><span class="mw-page-title-main">Data parallelism</span></span>

</h1>

</header>

<p><b>Data parallelism</b> is parallelization across multiple processors in <a href="Parallel_computing" title="Parallel computing">parallel computing</a> environments. It focuses on distributing the data across different nodes, which operate on the data in parallel. It can be applied on regular data structures like arrays and matrices by working on each element in parallel. It contrasts to <a href="Task_parallelism" title="Task parallelism">task parallelism</a> as another form of parallelism.

</p><p>A data parallel job on an array of <i>n</i> elements can be divided equally among all the processors. Let us assume we want to sum all the elements of the given array and the time for a single addition operation is Ta time units. In the case of sequential execution, the time taken by the process will be <i>n</i>×Ta time units as it sums up all the elements of an array. On the other hand, if we execute this job as a data parallel job on 4 processors the time taken would reduce to (<i>n</i>/4)×Ta + merging overhead time units. Parallel execution results in a speedup of 4 over sequential execution. The <a href="Locality_of_reference" title="Locality of reference">locality of data references</a> plays an important part in evaluating the performance of a data parallel programming model. Locality of data depends on the memory accesses performed by the program as well as the size of the cache.

</p>

<div class="mw-heading mw-heading2"><h2 id="History">History</h2></div>

<p>Exploitation of the concept of data parallelism started in 1960s with the development of the Solomon machine.<sup id="cite_ref-1" class="reference"><a href="#cite_note-1"><span class="cite-bracket">[</span>1<span class="cite-bracket">]</span></a></sup> The Solomon machine, also called a <a href="Vector_processor" title="Vector processor">vector processor</a>, was developed to expedite the performance of mathematical operations by working on a large data array (operating on multiple data in consecutive time steps). <a href="Concurrency_(computer_science)" title="Concurrency (computer science)">Concurrency</a> of data operations was also exploited by operating on multiple data at the same time using a single instruction. These processors were called 'array processors'.<sup id="cite_ref-2" class="reference"><a href="#cite_note-2"><span class="cite-bracket">[</span>2<span class="cite-bracket">]</span></a></sup> In the 1980s, the term was introduced <sup id="cite_ref-3" class="reference"><a href="#cite_note-3"><span class="cite-bracket">[</span>3<span class="cite-bracket">]</span></a></sup> to describe this programming style, which was widely used to program <a href="Connection_Machine" title="Connection Machine">Connection Machines</a> in data parallel languages like <a href="C*" title="C*">C*</a>. Today, data parallelism is best exemplified in <a href="Graphics_processing_unit" title="Graphics processing unit">graphics processing units</a> (GPUs), which use both the techniques of operating on multiple data in space and time using a single instruction.

</p><p>Most data parallel hardware supports only a fixed number of parallel levels, often only one. This means that within a parallel operation it is not possible to launch more parallel operations recursively, and means that programmers cannot make use of nested hardware parallelism. The programming language <a href="NESL" title="NESL">NESL</a> was an early effort at implementing a nested data-parallel programming model on flat parallel machines, and in particular introduced the <a href="Flattening_transformation" title="Flattening transformation">flattening transformation</a> that transforms nested data parallelism to flat data parallelism. This work was continued by other languages such as Data Parallel Haskell and <a href="Futhark_(programming_language)" title="Futhark (programming language)">Futhark</a>, although arbitrary nested data parallelism is not widely available in current data-parallel programming languages.

</p>

<div class="mw-heading mw-heading2"><h2 id="Description">Description</h2></div>

<p>In a multiprocessor system executing a single set of instructions (<a href="SIMD" class="mw-redirect" title="SIMD">SIMD</a>), data parallelism is achieved when each processor performs the same task on different distributed data. In some situations, a single execution thread controls operations on all the data. In others, different threads control the operation, but they execute the same code.

</p><p>For instance, consider <a href="Matrix_multiplication" title="Matrix multiplication">matrix multiplication</a> and addition in a sequential manner as discussed in the example.

</p>

<div class="mw-heading mw-heading2"><h2 id="Example">Example</h2></div>

<p>Below is the sequential pseudo-code for multiplication and addition of two matrices where the result is stored in the matrix <var style="padding-right: 1px;">C</var>. The pseudo-code for multiplication calculates the <a href="Dot_product" title="Dot product">dot product</a> of two matrices <var style="padding-right: 1px;">A</var>, <var style="padding-right: 1px;">B</var> and stores the result into the output matrix <var style="padding-right: 1px;">C</var>.

</p><p>If the following programs were executed sequentially, the time taken to calculate the result would be of the <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle O(n^{3})}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle O(n^{3})}</annotation>

</semantics>

</math></span><img src="./6b04f5c5cfea38f43406d9442387ad28555e2609.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:6.032ex; height:3.176ex;" alt="{\displaystyle O(n^{3})}" loading="lazy"></span>(assuming row lengths and column lengths of both matrices are n) and <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle O(n)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle O(n)}</annotation>

</semantics>

</p>

<div class="mw-highlight mw-highlight-lang-c mw-content-ltr" dir="ltr"><pre><span class="c1">// Matrix multiplication</span>

<span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">row_length_A</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="o">++</span><span class="p">)</span>

<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">k</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">k</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">column_length_B</span><span class="p">;</span><span class="w"> </span><span class="n">k</span><span class="o">++</span><span class="p">)</span>

<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">j</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">j</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">column_length_A</span><span class="p">;</span><span class="w"> </span><span class="n">j</span><span class="o">++</span><span class="p">)</span>

</pre></div>

<div class="mw-highlight mw-highlight-lang-c mw-content-ltr" dir="ltr"><pre><span class="c1">// Array addition</span>

</pre></div><p>We can exploit data parallelism in the preceding code to execute it faster as the arithmetic is loop independent. Parallelization of the matrix multiplication code is achieved by using <a href="OpenMP" title="OpenMP">OpenMP</a>. An OpenMP directive, "omp parallel for" instructs the compiler to execute the code in the for loop in parallel. For multiplication, we can divide matrix A and B into blocks along rows and columns respectively. This allows us to calculate every element in matrix C individually thereby making the task parallel. For example: <i>A[m x n] dot B [n x k]</i> can be finished in <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle O(n)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle O(n)}</annotation>

</semantics>

</math></span><img src="./34109fe397fdcff370079185bfdb65826cb5565a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:4.977ex; height:2.843ex;" alt="{\displaystyle O(n)}" loading="lazy"></span> instead of <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle O(m*n*k)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle O(m*n*k)}</annotation>

</semantics>

</math></span><img src="./49324edc2a64f04db3fdf3d6947b8c97db2aac7a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:12.618ex; height:2.843ex;" alt="{\displaystyle O(m*n*k)}" loading="lazy"></span> when executed in parallel using <i>m*k</i> processors.

</p>

<div class="mw-highlight mw-highlight-lang-c mw-content-ltr" dir="ltr"><pre><span class="c1">// Matrix multiplication in parallel</span>

<span class="cp">#pragma omp parallel for schedule(dynamic,1) collapse(2)</span>

</pre></div><p>It can be observed from the example that a lot of processors will be required as the matrix sizes keep on increasing. Keeping the execution time low is the priority but as the matrix size increases, we are faced with other constraints like complexity of such a system and its associated costs. Therefore, constraining the number of processors in the system, we can still apply the same principle and divide the data into bigger chunks to calculate the product of two matrices.<sup id="cite_ref-4" class="reference"><a href="#cite_note-4"><span class="cite-bracket">[</span>4<span class="cite-bracket">]</span></a></sup>

</p><p>For addition of arrays in a data parallel implementation, let's assume a more modest system with two <a href="Central_processing_unit" title="Central processing unit">central processing units</a> (CPU) A and B, CPU A could add all elements from the top half of the arrays, while CPU B could add all elements from the bottom half of the arrays. Since the two processors work in parallel, the job of performing array addition would take one half the time of performing the same operation in serial using one CPU alone.

</p><p>The program expressed in <a href="Pseudocode" title="Pseudocode">pseudocode</a> below—which applies some arbitrary operation, <code>foo</code>, on every element in the array <code>d</code>—illustrates data parallelism:<sup id="cite_ref-5" class="reference"><a href="#cite_note-5"><span class="cite-bracket">[</span>nb 1<span class="cite-bracket">]</span></a></sup>

</p>

lower_limit := 1

upper_limit := round(d.length / 2)

lower_limit := round(d.length / 2) + 1

upper_limit := d.length

<b>for</b> i from lower_limit to upper_limit by 1 <b>do</b>

foo(d[i])

</pre>

<p>In an <a href="SPMD" class="mw-redirect" title="SPMD">SPMD</a> system executed on 2 processor system, both CPUs will execute the code.

</p><p>Data parallelism emphasizes the distributed (parallel) nature of the data, as opposed to the processing (task parallelism). Most real programs fall somewhere on a continuum between task parallelism and data parallelism.

</p>

<div class="mw-heading mw-heading2"><h2 id="Steps_to_parallelization">Steps to parallelization</h2></div>

<p>The process of parallelizing a sequential program can be broken down into four discrete steps.<sup id="cite_ref-Solihin_6-0" class="reference"><a href="#cite_note-Solihin-6"><span class="cite-bracket">[</span>5<span class="cite-bracket">]</span></a></sup>

</p>

<th>Type

</th>

<th>Description

</th></tr>

<tr>

<td>Decomposition

</td>

<td>The program is broken down into tasks, the smallest exploitable unit of concurrence.

</td></tr>

<tr>

<td>Assignment

</td>

<td>Tasks are assigned to processes.

</td></tr>

<tr>

<td>Orchestration

</td>

<td>Data access, communication, and synchronization of processes.

</td></tr>

<tr>

<td>Mapping

</td>

<td>Processes are bound to processors.

</td></tr></tbody></table>

<div class="mw-heading mw-heading2"><h2 id="Data_parallelism_vs._task_parallelism">Data parallelism vs. task parallelism</h2></div>

<th>Data parallelism

</th>

<th>Task parallelism

</th></tr>

<tr>

<td>Same operations are performed on different subsets of same data.

</td>

<td>Different operations are performed on the same or different data.

</td></tr>

<tr>

<td>Synchronous computation

</td>

<td>Asynchronous computation

</td></tr>

<tr>

<td>Speedup is more as there is only one execution thread operating on all sets of data.

</td>

<td>Speedup is less as each processor will execute a different thread or process on the same or different set of data.

</td></tr>

<tr>

<td>Amount of parallelization is proportional to the input data size.

</td>

<td>Amount of parallelization is proportional to the number of independent tasks to be performed.

</td></tr>

<tr>

<td>Designed for optimum <a href="Load_balancing_(computing)" title="Load balancing (computing)">load balance</a> on multi processor system.

</td>

<td>Load balancing depends on the availability of the hardware and scheduling algorithms like static and dynamic scheduling.

</td></tr></tbody></table>

<div class="mw-heading mw-heading2"><h2 id="Data_parallelism_vs._model_parallelism">Data parallelism vs. model parallelism</h2></div>

<th>Data parallelism

</th>

<th>Model parallelism

</th></tr>

<tr>

<td>Same model is used for every thread but the data given to each of them is divided and shared.

</td>

<td>Same data is used for every thread, and model is split among threads.

</td></tr>

<tr>

<td>It is fast for small networks but very slow for large networks since large amounts of data needs to be transferred between processors all at once.

</td>

<td>It is slow for small networks and fast for large networks.

</td></tr>

<tr>

<td>Data parallelism is ideally used in array and matrix computations and convolutional neural networks

</td>

<td>Model parallelism finds its applications in deep learning

</td></tr></tbody></table>

</p>

<div class="mw-heading mw-heading2"><h2 id="Mixed_data_and_task_parallelism">Mixed data and task parallelism</h2></div>

<p>Data and task parallelism, can be simultaneously implemented by combining them together for the same application. This is called Mixed data and task parallelism. Mixed parallelism requires sophisticated scheduling algorithms and software support. It is the best kind of parallelism when communication is slow and number of processors is large.<sup id="cite_ref-8" class="reference"><a href="#cite_note-8"><span class="cite-bracket">[</span>7<span class="cite-bracket">]</span></a></sup>

</p><p>Mixed data and task parallelism has many applications. It is particularly used in the following applications:

</p>

<ol><li>Mixed data and task parallelism finds applications in the global climate modeling. Large data parallel computations are performed by creating grids of data representing Earth's atmosphere and oceans and task parallelism is employed for simulating the function and model of the physical processes.</li>

<li>In timing based <a href="Circuit_simulation" class="mw-redirect" title="Circuit simulation">circuit simulation</a>. The data is divided among different sub-circuits and parallelism is achieved with orchestration from the tasks.</li></ol>

<div class="mw-heading mw-heading2"><h2 id="Data_parallel_programming_environments">Data parallel programming environments</h2></div>

<p>A variety of data parallel programming environments are available today, most widely used of which are:

</p>

<ol><li><a href="Message_Passing_Interface" title="Message Passing Interface">Message Passing Interface</a>: It is a cross-platform message passing programming interface for parallel computers. It defines the semantics of library functions to allow users to write portable message passing programs in C, C++ and Fortran.</li>

<li><a href="OpenMP" title="OpenMP">OpenMP</a>:<sup id="cite_ref-9" class="reference"><a href="#cite_note-9"><span class="cite-bracket">[</span>8<span class="cite-bracket">]</span></a></sup> It's an Application Programming Interface (API) which supports <a href="Shared_memory" title="Shared memory">shared memory</a> programming models on multiple platforms of multiprocessor systems. Since version 4.5, OpenMP is also able to target devices other than typical CPUs. It can program FPGAs, DSPs, GPUs and more. It is not confined to GPUs like OpenACC.</li>

<li><a href="CUDA" title="CUDA">CUDA</a> and <a href="OpenACC" title="OpenACC">OpenACC</a>: CUDA and OpenACC (respectively) are parallel computing API platforms designed to allow a software engineer to utilize GPUs' computational units for general purpose processing.</li>

<li><a href="Threading_Building_Blocks" title="Threading Building Blocks">Threading Building Blocks</a> and <a href="RaftLib" title="RaftLib">RaftLib</a>: Both open source programming environments that enable mixed data/task parallelism in C/C++ environments across heterogeneous resources.</li></ol>

<div class="mw-heading mw-heading2"><h2 id="Applications">Applications</h2></div>

<p>Data parallelism finds its applications in a variety of fields ranging from physics, chemistry, biology, material sciences to signal processing. Sciences imply data parallelism for simulating models like molecular dynamics,<sup id="cite_ref-10" class="reference"><a href="#cite_note-10"><span class="cite-bracket">[</span>9<span class="cite-bracket">]</span></a></sup> sequence analysis of genome data <sup id="cite_ref-11" class="reference"><a href="#cite_note-11"><span class="cite-bracket">[</span>10<span class="cite-bracket">]</span></a></sup> and other physical phenomenon. Driving forces in signal processing for data parallelism are video encoding, image and graphics processing, wireless communications <sup id="cite_ref-12" class="reference"><a href="#cite_note-12"><span class="cite-bracket">[</span>11<span class="cite-bracket">]</span></a></sup> to name a few.

</p>

<div class="mw-heading mw-heading3"><h3 id="Data-intensive_computing">Data-intensive computing</h3></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .excerpt-hat .mw-editsection-like{font-style:normal}

/* end https://en.wikipedia.org/ */

</style><style data-mw-deduplicate="TemplateStyles:r1236090951">

/* start https://en.wikipedia.org/ */

.mw-parser-output .hatnote{font-style:italic}.mw-parser-output div.hatnote{padding-left:1.6em;margin-bottom:0.5em}.mw-parser-output .hatnote i{font-style:normal}.mw-parser-output .hatnote+link+.hatnote{margin-top:-0.5em}@media print{body.ns-0 .mw-parser-output .hatnote{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div role="note" class="hatnote navigation-not-searchable dablink excerpt-hat selfref">This section is an excerpt from <a href="Data-intensive_computing" title="Data-intensive computing">Data-intensive computing</a>.<span class="mw-editsection-like "><span class="mw-editsection-bracket">[</span><a class="external text external" href="https://en.wikipedia.org/w/index.php?title=Data-intensive_computing&action=edit">edit</a><span class="mw-editsection-bracket">]</span></span></div><div class="excerpt">

<a href="Data-intensive_computing" title="Data-intensive computing">Data-intensive computing</a> is a class of <a href="Parallel_computing" title="Parallel computing">parallel computing</a> applications which use a <a href="Data_parallel" class="mw-redirect" title="Data parallel">data parallel</a> approach to process large volumes of data typically <a href="Terabytes" class="mw-redirect" title="Terabytes">terabytes</a> or <a href="Petabytes" class="mw-redirect" title="Petabytes">petabytes</a> in size and typically referred to as <a href="Big_data" title="Big data">big data</a>. Computing applications that devote most of their execution time to computational requirements are deemed compute-intensive, whereas applications are deemed data-intensive if they require large volumes of data and devote most of their processing time to <a href="Input/output" title="Input/output">input/output</a> and manipulation of data.<sup id="cite_ref-13" class="reference"><a href="#cite_note-13"><span class="cite-bracket">[</span>12<span class="cite-bracket">]</span></a></sup></div></div>

<ul><li><a href="Active_message" title="Active message">Active message</a></li>

<li><a href="Instruction_level_parallelism" class="mw-redirect" title="Instruction level parallelism">Instruction level parallelism</a></li>

<li><a href="Parallel_programming_model" title="Parallel programming model">Parallel programming model</a></li>

<li><a href="Prefix_sum" title="Prefix sum">Prefix sum</a></li>

<li><a href="Scalable_parallelism" title="Scalable parallelism">Scalable parallelism</a></li>

<li><a href="Segmented_scan" title="Segmented scan">Segmented scan</a></li>

<li><a href="Thread_level_parallelism" class="mw-redirect" title="Thread level parallelism">Thread level parallelism</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="Notes">Notes</h2></div>

<li id="cite_note-5"><span class="mw-cite-backlink"><b><a href="#cite_ref-5">^</a></b></span> <span class="reference-text">Some input data (e.g. when <code>d.length</code> evaluates to 1 and <code>round</code> rounds towards zero [this is just an example, there are no requirements on what type of rounding is used]) will lead to <code>lower_limit</code> being greater than <code>upper_limit</code>, it's assumed that the loop will exit immediately (i.e. zero iterations will occur) when this happens.</span>

</li>

</ol></div>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.computer.org/csdl/pds/api/csdl/proceedings/download-article/50610097/pdf?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJjc2RsX2FwaSIsImF1ZCI6ImNzZGxfYXBpX2Rvd25sb2FkX3Rva2VuIiwic3ViIjoiYW5vbnltb3VzQGNvbXB1dGVyLm9yZyIsImVtYWlsIjoiYW5vbnltb3VzQGNvbXB1dGVyLm9yZyIsImV4cCI6MTU2NTEwNTAxMX0.AD74lJbBAdGWNvVIpeeTmyF1S7hb4_rUDeSeoDoJ0R4">"The Solomon Computer"</a>.</cite></span>

</li>

<li id="cite_note-2"><span class="mw-cite-backlink"><b><a href="#cite_ref-2">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.ece.cmu.edu/~ece740/f13/lib/exe/fetch.php%3Fmedia%3Dseth-740-fall13-module5.1-simd-vector-gpu.pdf">"SIMD/Vector/GPU"</a> <span class="cs1-format">(PDF)</span><span class="reference-accessdate">. Retrieved <span class="nowrap">2016-09-07</span></span>.</cite></span>

</li>

<li id="cite_note-3"><span class="mw-cite-backlink"><b><a href="#cite_ref-3">^</a></b></span> <span class="reference-text"><a href="Daniel_Hillis" class="mw-redirect" title="Daniel Hillis">Hillis, W. Daniel</a> and <a href="Guy_Steele" class="mw-redirect" title="Guy Steele">Steele, Guy L.</a>, <a rel="nofollow" class="external text" href="https://dx.doi.org/10.1145/7902.7903"><cite>Data Parallel Algorithms</cite></a> <a href="Communications_of_the_ACM" title="Communications of the ACM">Communications of the ACM</a>December 1986</span>

</li>

<li id="cite_note-4"><span class="mw-cite-backlink"><b><a href="#cite_ref-4">^</a></b></span> <span class="reference-text"><cite id="CITEREFBarney" class="citation web cs1">Barney, Blaise. <a rel="nofollow" class="external text" href="https://web.archive.org/web/20130610122229/https://computing.llnl.gov/tutorials/parallel_comp/">"Introduction to Parallel Computing"</a>. <i>computing.llnl.gov</i>. Archived from <a rel="nofollow" class="external text" href="https://computing.llnl.gov/tutorials/parallel_comp/">the original</a> on 2013-06-10<span class="reference-accessdate">. Retrieved <span class="nowrap">2016-09-07</span></span>.</cite></span>

</li>

<li id="cite_note-Solihin-6"><span class="mw-cite-backlink"><b><a href="#cite_ref-Solihin_6-0">^</a></b></span> <span class="reference-text"><cite id="CITEREFSolihin2016" class="citation book cs1">Solihin, Yan (2016). <i>Fundamentals of Parallel Architecture</i>. Boca Raton, FL: CRC Press. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1-4822-1118-4</bdi>.</cite></span>

</li>

<li id="cite_note-7"><span class="mw-cite-backlink"><b><a href="#cite_ref-7">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="http://timdettmers.com/2014/11/09/model-parallelism-deep-learning/">"How to Parallelize Deep Learning on GPUs Part 2/2: Model Parallelism"</a>. <i>Tim Dettmers</i>. 2014-11-09<span class="reference-accessdate">. Retrieved <span class="nowrap">2016-09-13</span></span>.</cite></span>

</li>

<li id="cite_note-8"><span class="mw-cite-backlink"><b><a href="#cite_ref-8">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="http://www.netlib.org/lapack/lawnspdf/lawn97.pdf">"The Netlib"</a> <span class="cs1-format">(PDF)</span>.</cite></span>

</li>

<li id="cite_note-9"><span class="mw-cite-backlink"><b><a href="#cite_ref-9">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://web.archive.org/web/20160905232633/http://openmp.org/wp/">"OpenMP.org"</a>. <i>openmp.org</i>. Archived from <a rel="nofollow" class="external text" href="http://openmp.org/wp/">the original</a> on 2016-09-05<span class="reference-accessdate">. Retrieved <span class="nowrap">2016-09-07</span></span>.</cite></span>

</li>

<li id="cite_note-10"><span class="mw-cite-backlink"><b><a href="#cite_ref-10">^</a></b></span> <span class="reference-text"><cite id="CITEREFBoyerPawley1988" class="citation journal cs1">Boyer, L. L; Pawley, G. S (1988-10-01). "Molecular dynamics of clusters of particles interacting with pairwise forces using a massively parallel computer". <i>Journal of Computational Physics</i>. <b>78</b> (2): <span class="nowrap">405–</span>423. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/1988JCoPh..78..405B">1988JCoPh..78..405B</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1016%2F0021-9991%2888%2990057-5">10.1016/0021-9991(88)90057-5</a>.</cite></span>

</li>

<li id="cite_note-11"><span class="mw-cite-backlink"><b><a href="#cite_ref-11">^</a></b></span> <span class="reference-text"><cite id="CITEREFYapFriederMartino1998" class="citation journal cs1">Yap, T.K.; Frieder, O.; Martino, R.L. (1998). "Parallel computation in biological sequence analysis". <i>IEEE Transactions on Parallel and Distributed Systems</i>. <b>9</b> (3): <span class="nowrap">283–</span>294. <a href="CiteSeerX_(identifier)" class="mw-redirect" title="CiteSeerX (identifier)">CiteSeerX</a> <span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.30.2819">10.1.1.30.2819</a></span>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2F71.674320">10.1109/71.674320</a>.</cite></span>

</li>

<li id="cite_note-12"><span class="mw-cite-backlink"><b><a href="#cite_ref-12">^</a></b></span> <span class="reference-text"><cite id="CITEREFSinghLeeLuKurdahi2000" class="citation journal cs1">Singh, H.; Lee, Ming-Hau; Lu, Guangming; Kurdahi, F.J.; Bagherzadeh, N.; Filho, E.M. Chaves (2000-06-01). <a rel="nofollow" class="external text" href="https://www.researchgate.net/publication/3044209">"MorphoSys: an integrated reconfigurable system for data-parallel and computation-intensive applications"</a>. <i>IEEE Transactions on Computers</i>. <b>49</b> (5): <span class="nowrap">465–</span>481. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2F12.859540">10.1109/12.859540</a>. <a href="ISSN_(identifier)" class="mw-redirect" title="ISSN (identifier)">ISSN</a> <a rel="nofollow" class="external text" href="https://search.worldcat.org/issn/0018-9340">0018-9340</a>.</cite></span>

</li>

<li id="cite_note-13"><span class="mw-cite-backlink"><b><a href="#cite_ref-13">^</a></b></span> <span class="reference-text"><a rel="nofollow" class="external text" href="http://www.cse.fau.edu/~borko/HandbookofCloudComputing.html">Handbook of Cloud Computing</a>, "Data-Intensive Technologies for Cloud Computing," by A.M. Middleton. Handbook of Cloud Computing. Springer, 2010.</span>

</li>

</ol></div></div>

<ul><li><a href="Daniel_Hillis" class="mw-redirect" title="Daniel Hillis">Hillis, W. Daniel</a> and <a href="Guy_Steele" class="mw-redirect" title="Guy Steele">Steele, Guy L.</a>, <a rel="nofollow" class="external text" href="https://dx.doi.org/10.1145/7902.7903"><cite>Data Parallel Algorithms</cite></a> <a href="Communications_of_the_ACM" title="Communications of the ACM">Communications of the ACM</a> December 1986</li>

<li>Blelloch, Guy E, <cite>Vector Models for Data-Parallel Computing</cite> MIT Press 1990. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>0-262-02313-X</bdi></li></ul>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hlist dl,.mw-parser-output .hlist ol,.mw-parser-output .hlist ul{margin:0;padding:0}.mw-parser-output .hlist dd,.mw-parser-output .hlist dt,.mw-parser-output .hlist li{margin:0;display:inline}.mw-parser-output .hlist.inline,.mw-parser-output .hlist.inline dl,.mw-parser-output .hlist.inline ol,.mw-parser-output .hlist.inline ul,.mw-parser-output .hlist dl dl,.mw-parser-output .hlist dl ol,.mw-parser-output .hlist dl ul,.mw-parser-output .hlist ol dl,.mw-parser-output .hlist ol ol,.mw-parser-output .hlist ol ul,.mw-parser-output .hlist ul dl,.mw-parser-output .hlist ul ol,.mw-parser-output .hlist ul ul{display:inline}.mw-parser-output .hlist .mw-empty-li{display:none}.mw-parser-output .hlist dt::after{content:": "}.mw-parser-output .hlist dd::after,.mw-parser-output .hlist li::after{content:" · ";font-weight:bold}.mw-parser-output .hlist dd:last-child::after,.mw-parser-output .hlist dt:last-child::after,.mw-parser-output .hlist li:last-child::after{content:none}.mw-parser-output .hlist dd dd:first-child::before,.mw-parser-output .hlist dd dt:first-child::before,.mw-parser-output .hlist dd li:first-child::before,.mw-parser-output .hlist dt dd:first-child::before,.mw-parser-output .hlist dt dt:first-child::before,.mw-parser-output .hlist dt li:first-child::before,.mw-parser-output .hlist li dd:first-child::before,.mw-parser-output .hlist li dt:first-child::before,.mw-parser-output .hlist li li:first-child::before{content:" (";font-weight:normal}.mw-parser-output .hlist dd dd:last-child::after,.mw-parser-output .hlist dd dt:last-child::after,.mw-parser-output .hlist dd li:last-child::after,.mw-parser-output .hlist dt dd:last-child::after,.mw-parser-output .hlist dt dt:last-child::after,.mw-parser-output .hlist dt li:last-child::after,.mw-parser-output .hlist li dd:last-child::after,.mw-parser-output .hlist li dt:last-child::after,.mw-parser-output .hlist li li:last-child::after{content:")";font-weight:normal}.mw-parser-output .hlist ol{counter-reset:listitem}.mw-parser-output .hlist ol>li{counter-increment:listitem}.mw-parser-output .hlist ol>li::before{content:" "counter(listitem)"\a0 "}.mw-parser-output .hlist dd ol>li:first-child::before,.mw-parser-output .hlist dt ol>li:first-child::before,.mw-parser-output .hlist li ol>li:first-child::before{content:" ("counter(listitem)"\a0 "}

/* end https://en.wikipedia.org/ */

</style><style data-mw-deduplicate="TemplateStyles:r1236075235">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbox{box-sizing:border-box;border:1px solid #a2a9b1;width:100%;clear:both;font-size:88%;text-align:center;padding:1px;margin:1em auto 0}.mw-parser-output .navbox .navbox{margin-top:0}.mw-parser-output .navbox+.navbox,.mw-parser-output .navbox+.navbox-styles+.navbox{margin-top:-1px}.mw-parser-output .navbox-inner,.mw-parser-output .navbox-subgroup{width:100%}.mw-parser-output .navbox-group,.mw-parser-output .navbox-title,.mw-parser-output .navbox-abovebelow{padding:0.25em 1em;line-height:1.5em;text-align:center}.mw-parser-output .navbox-group{white-space:nowrap;text-align:right}.mw-parser-output .navbox,.mw-parser-output .navbox-subgroup{background-color:#fdfdfd}.mw-parser-output .navbox-list{line-height:1.5em;border-color:#fdfdfd}.mw-parser-output .navbox-list-with-group{text-align:left;border-left-width:2px;border-left-style:solid}.mw-parser-output tr+tr>.navbox-abovebelow,.mw-parser-output tr+tr>.navbox-group,.mw-parser-output tr+tr>.navbox-image,.mw-parser-output tr+tr>.navbox-list{border-top:2px solid #fdfdfd}.mw-parser-output .navbox-title{background-color:#ccf}.mw-parser-output .navbox-abovebelow,.mw-parser-output .navbox-group,.mw-parser-output .navbox-subgroup .navbox-title{background-color:#ddf}.mw-parser-output .navbox-subgroup .navbox-group,.mw-parser-output .navbox-subgroup .navbox-abovebelow{background-color:#e6e6ff}.mw-parser-output .navbox-even{background-color:#f7f7f7}.mw-parser-output .navbox-odd{background-color:transparent}.mw-parser-output .navbox .hlist td dl,.mw-parser-output .navbox .hlist td ol,.mw-parser-output .navbox .hlist td ul,.mw-parser-output .navbox td.hlist dl,.mw-parser-output .navbox td.hlist ol,.mw-parser-output .navbox td.hlist ul{padding:0.125em 0}.mw-parser-output .navbox .navbar{display:block;font-size:100%}.mw-parser-output .navbox-title .navbar{float:left;text-align:left;margin-right:0.5em}body.skin--responsive .mw-parser-output .navbox-image img{max-width:none!important}@media print{body.ns-0 .mw-parser-output .navbox{display:none!important}}

/* end https://en.wikipedia.org/ */

</style></div><div role="navigation" class="navbox" aria-labelledby="Parallel_computing346" style="padding:3px"><table class="nowraplinks hlist mw-collapsible autocollapse navbox-inner" style="border-spacing:0;background:transparent;color:inherit"><tbody><tr><th scope="col" class="navbox-title" colspan="2"><style data-mw-deduplicate="TemplateStyles:r1239400231">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbar{display:inline;font-size:88%;font-weight:normal}.mw-parser-output .navbar-collapse{float:left;text-align:left}.mw-parser-output .navbar-boxtext{word-spacing:0}.mw-parser-output .navbar ul{display:inline-block;white-space:nowrap;line-height:inherit}.mw-parser-output .navbar-brackets::before{margin-right:-0.125em;content:"[ "}.mw-parser-output .navbar-brackets::after{margin-left:-0.125em;content:" ]"}.mw-parser-output .navbar li{word-spacing:-0.125em}.mw-parser-output .navbar a>span,.mw-parser-output .navbar a>abbr{text-decoration:inherit}.mw-parser-output .navbar-mini abbr{font-variant:small-caps;border-bottom:none;text-decoration:none;cursor:inherit}.mw-parser-output .navbar-ct-full{font-size:114%;margin:0 7em}.mw-parser-output .navbar-ct-mini{font-size:114%;margin:0 4em}html.skin-theme-clientpref-night .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}@media(prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}}@media print{.mw-parser-output .navbar{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div id="Parallel_computing346" style="font-size:114%;margin:0 4em"><a href="Parallel_computing" title="Parallel computing">Parallel computing</a></div></th></tr><tr><th scope="row" class="navbox-group" style="width:1%">General</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Distributed_computing" title="Distributed computing">Distributed computing</a></li>

<li><a href="Parallel_computing" title="Parallel computing">Parallel computing</a></li>

<li><a href="Parallel_algorithm" title="Parallel algorithm">Parallel algorithm</a></li>

<li><a href="Massively_parallel" title="Massively parallel">Massively parallel</a></li>

<li><a href="Cloud_computing" title="Cloud computing">Cloud computing</a></li>

<li><a href="High-performance_computing" title="High-performance computing">High-performance computing</a></li>

<li><a href="Multiprocessing" title="Multiprocessing">Multiprocessing</a></li>

<li><a href="Manycore_processor" title="Manycore processor">Manycore processor</a></li>

<li><a href="General-purpose_computing_on_graphics_processing_units" title="General-purpose computing on graphics processing units">GPGPU</a></li>

<li><a href="Computer_network" title="Computer network">Computer network</a></li>

<li><a href="Systolic_array" title="Systolic array">Systolic array</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Levels</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<li><a href="Instruction-level_parallelism" title="Instruction-level parallelism">Instruction</a></li>

<li><a href="Task_parallelism" title="Task parallelism">Thread</a></li>

<li><a href="Memory-level_parallelism" title="Memory-level parallelism">Memory</a></li>

<li><a href="Pipeline_(computing)" title="Pipeline (computing)">Pipeline</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Multithreading_(computer_architecture)" title="Multithreading (computer architecture)">Multithreading</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Temporal_multithreading" title="Temporal multithreading">Temporal</a></li>

<li><a href="Simultaneous_multithreading" title="Simultaneous multithreading">Simultaneous</a> (SMT)</li>

<li><a href="Simultaneous_and_heterogeneous_multithreading" title="Simultaneous and heterogeneous multithreading">Simultaneous and heterogenous</a></li>

<li><a href="Speculative_multithreading" title="Speculative multithreading">Speculative</a> (SpMT)</li>

<li><a href="Preemption_(computing)" title="Preemption (computing)">Preemptive</a></li>

<li><a href="Computer_multitasking#Cooperative_multitasking" title="Computer multitasking">Cooperative</a></li>

<li><a href="Bulldozer_(microarchitecture)#Bulldozer_core" title="Bulldozer (microarchitecture)">Clustered multi-thread</a> (CMT)</li>

<li><a href="Hardware_scout" title="Hardware scout">Hardware scout</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Theory</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Parallel_RAM" title="Parallel RAM">PRAM model</a></li>

<li><a href="Parallel_external_memory" title="Parallel external memory">PEM model</a></li>

<li><a href="Analysis_of_parallel_algorithms" title="Analysis of parallel algorithms">Analysis of parallel algorithms</a></li>

<li><a href="Amdahl's_law" title="Amdahl's law">Amdahl's law</a></li>

<li><a href="Gustafson's_law" title="Gustafson's law">Gustafson's law</a></li>

<li><a href="Cost_efficiency" title="Cost efficiency">Cost efficiency</a></li>

<li><a href="Karp%E2%80%93Flatt_metric" title="Karp–Flatt metric">Karp–Flatt metric</a></li>

<li><a href="Parallel_slowdown" title="Parallel slowdown">Slowdown</a></li>

<li><a href="Speedup" title="Speedup">Speedup</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Elements</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Process_(computing)" title="Process (computing)">Process</a></li>

<li><a href="Thread_(computing)" title="Thread (computing)">Thread</a></li>

<li><a href="Fiber_(computer_science)" title="Fiber (computer science)">Fiber</a></li>

<li><a href="Instruction_window" title="Instruction window">Instruction window</a></li>

<li><a href="Array_(data_structure)" title="Array (data structure)">Array</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Coordination</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Multiprocessing" title="Multiprocessing">Multiprocessing</a></li>

<li><a href="Memory_coherence" title="Memory coherence">Memory coherence</a></li>

<li><a href="Cache_coherence" title="Cache coherence">Cache coherence</a></li>

<li><a href="Cache_invalidation" title="Cache invalidation">Cache invalidation</a></li>

<li><a href="Barrier_(computer_science)" title="Barrier (computer science)">Barrier</a></li>

<li><a href="Synchronization_(computer_science)" title="Synchronization (computer science)">Synchronization</a></li>

<li><a href="Application_checkpointing" title="Application checkpointing">Application checkpointing</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Computer_programming" title="Computer programming">Programming</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Stream_processing" title="Stream processing">Stream processing</a></li>

<li><a href="Dataflow_programming" title="Dataflow programming">Dataflow programming</a></li>

<li><a href="Parallel_programming_model" title="Parallel programming model">Models</a>

<ul><li><a href="Implicit_parallelism" title="Implicit parallelism">Implicit parallelism</a></li>

<li><a href="Explicit_parallelism" title="Explicit parallelism">Explicit parallelism</a></li>

<li><a href="Concurrency_(computer_science)" title="Concurrency (computer science)">Concurrency</a></li></ul></li>

<li><a href="Non-blocking_algorithm" title="Non-blocking algorithm">Non-blocking algorithm</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Computer_hardware" title="Computer hardware">Hardware</a></th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Flynn's_taxonomy" title="Flynn's taxonomy">Flynn's taxonomy</a>

<ul><li><a href="Single_instruction%2C_multiple_threads" title="Single instruction, multiple threads">Array processing</a> (SIMT)</li>

<li><a href="Flynn's_taxonomy#Pipelined_processor" title="Flynn's taxonomy">Pipelined processing</a></li>

<li><a href="Flynn's_taxonomy#Associative_processor" title="Flynn's taxonomy">Associative processing</a></li></ul></li>

<li><a href="Dataflow_architecture" title="Dataflow architecture">Dataflow architecture</a></li>

<li><a href="Instruction_pipelining" title="Instruction pipelining">Pipelined processor</a></li>

<li><a href="Superscalar_processor" title="Superscalar processor">Superscalar processor</a></li>

<li><a href="Vector_processor" title="Vector processor">Vector processor</a></li>

<li><a href="Multiprocessing" title="Multiprocessing">Multiprocessor</a>

<ul><li><a href="Symmetric_multiprocessing" title="Symmetric multiprocessing">symmetric</a></li>

<li><a href="Asymmetric_multiprocessing" title="Asymmetric multiprocessing">asymmetric</a></li></ul></li>

<li><a href="Semiconductor_memory" title="Semiconductor memory">Memory</a>

<ul><li><a href="Shared_memory" title="Shared memory">shared</a></li>

<li><a href="Distributed_memory" title="Distributed memory">distributed</a></li>

<li><a href="Distributed_shared_memory" title="Distributed shared memory">distributed shared</a></li>

<li><a href="Massively_parallel" title="Massively parallel">Massively parallel</a> computer</li>

<li><a href="Computer_cluster" title="Computer cluster">Computer cluster</a>

<ul><li><a href="Beowulf_cluster" title="Beowulf cluster">Beowulf cluster</a></li></ul></li>

<li><a href="Grid_computing" title="Grid computing">Grid computer</a></li>

<li><a href="Hardware_acceleration" title="Hardware acceleration">Hardware acceleration</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="API" title="API">APIs</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Ateji_PX" title="Ateji PX">Ateji PX</a></li>

<li><a href="Boost_(C%2B%2B_libraries)" title="Boost (C++ libraries)">Boost</a></li>

<li><a href="Chapel_(programming_language)" title="Chapel (programming language)">Chapel</a></li>

<li><a href="Charm%2B%2B" title="Charm++">Charm++</a></li>

<li><a href="Coarray_Fortran" title="Coarray Fortran">Coarray Fortran</a></li>

<li><a href="Dryad_(programming)" title="Dryad (programming)">Dryad</a></li>

<li><a href="Global_Arrays" title="Global Arrays">Global Arrays</a></li>

<li><a href="GPUOpen" title="GPUOpen">GPUOpen</a></li>

<li><a href="OpenMP" title="OpenMP">OpenMP</a></li>

<li><a href="OpenCL" title="OpenCL">OpenCL</a></li>

<li><a href="OpenHMPP" title="OpenHMPP">OpenHMPP</a></li>

<li><a href="OpenACC" title="OpenACC">OpenACC</a></li>

<li><a href="Parallel_Extensions" title="Parallel Extensions">Parallel Extensions</a></li>

<li><a href="Pthreads" title="Pthreads">pthreads</a></li>

<li><a href="RaftLib" title="RaftLib">RaftLib</a></li>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Problems</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Automatic_parallelization" title="Automatic parallelization">Automatic parallelization</a></li>

<li><a href="Deadlock_(computer_science)" title="Deadlock (computer science)">Deadlock</a></li>

<li><a href="Deterministic_algorithm" title="Deterministic algorithm">Deterministic algorithm</a></li>

<li><a href="Embarrassingly_parallel" title="Embarrassingly parallel">Embarrassingly parallel</a></li>

<li><a href="Parallel_slowdown" title="Parallel slowdown">Parallel slowdown</a></li>

<li><a href="Race_condition" title="Race condition">Race condition</a></li>

<li><a href="Software_lockout" title="Software lockout">Software lockout</a></li>

<li><a href="Scalability" title="Scalability">Scalability</a></li>

<li><a href="Starvation_(computer_science)" title="Starvation (computer science)">Starvation</a></li></ul>

</div></td></tr><tr><td class="navbox-abovebelow" colspan="2"><div>

<ul><li><span class="noviewer" typeof="mw:File"><span title="Category"></span></span> Category: Parallel computing</li></ul>

</div></td></tr></tbody></table></div></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-03-25" href="https://en.wikipedia.org/wiki/?title=Data_parallelism&oldid=1282230741">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>